In the human-to-machine interaction domain, device controllers have been used for many years as interfaces between human users and hardware platforms. Through such interfaces, the user has been able to provide commands to a computerised system having means for receiving, processing, and executing actions in accordance with those commands. Device controllers may be wired or wireless and include mouse devices for computers, computer keyboards, TV remote controllers, game controllers, pads, virtual reality gloves . . . etc. All of these device controller systems share one common specification which is the use of known sensing devices, such as, for example, trigger buttons, multi-directional pads, inertial sensors units etc.
Furthermore, with the advance of technology in the virtual reality domain as well as in the consumer electronic domain, several systems have been developed that use such devices or a combination thereof. In particular, it has now become mandatory for the user to carry device controllers on him/her, wear or handle them in order to improve immersion into a virtual reality environment or in order to improve user experience for interactions with computerised systems.
For example, in US-A-2010/0304868, a multi-positional three-dimensional handled controller is described for interactivity with computerised systems. The controller provides control for a variety of applications and simulations, whilst providing an intuitive interface to interact with both two-dimensional and three-dimensional scenarios generated as a result of a computer program being executed through the computerised system of a game client. Such game clients can be a dedicated games console which can execute a locally stored program or can be connected to the internet so that a user can interact with another user at a different location. The controller is configured for use in multiple positions in order to provide flexibility in relation to the manipulation and handling of the controller. The controller comprises a handle having sensors that detect movement of input features relative to the handle, for example, a gyroscope, accelerometers, buttons, a joystick and a trackball. In one embodiment, the controller is hand-held and can operate in a “lollipop” mode where only the motion of the hand of a user provides wireless signals for interaction with an associated games console using a wireless protocol falling within the IEEE 802.11 specification. In another embodiment, the controller is again hand-held but operated in conjunction with a 2D camera that tracks the colour ball to detect gestures and which provides additional wireless signals to be processed by the games console. In a further embodiment, the controller can be operated with two hands. Ergonomics of the controller described in US-A-2010/0304868 require that the user actively and voluntary holds the controller with a hand and does not allow free natural movements, including especially fingers movements, for gesture recognition within a gesture-based interaction system.
In US-A-2011/0269544, a hand-held computer input device is described for interacting with a games console. The device comprises a body on which are formed protrusions, the fingers and thumb of the hand of the user engaging with respective one of the protrusions. Each protrusion is configured to detect movement relative to one or more of the fingers of the user. Movement of the hand of the user as well as his/her thumb and fingers are sensed by the games console and tactile feedback can be provided to the user in response to commands transmitted to the games console. A pair of devices may be utilised but these are handled so that only one device fits the right hand and the other device fits the left hand. The hand-held interactive device described in US-A-2011/0269544 may be considered to be ergonomic from the point of view of fitting to the hand and fingers of a user, but it requires the user to grip the device for operation and does not allow free movement of the hands and/or fingers for gesture recognition within a natural gesture-based computerised system.
US-A-2008/084385 describes a computer pointing device which is worn on the hand and is operated without needing to contact a surface. Sensors located within the computer pointing device sense the movements of the hand of the user and converts those signals into signals for a host computer. The signals can be used to move a cursor or the like and provide right and left mouse clicks as well as scrolling operations. The device can be activated and deactivated so that only intended movements are transmitted to the host computer.
In the article entitled “Providing Haptic Feedback Using the Kinect” by Brandon Shrewsbury, Dundee, Scotland, UK, 26 Oct. 2011, a glove having a wearable vibrotactile feedback system is proposed which utilises buzzer motors to alert the user of objects within the field of view of the Kinect system. The buzzer motors are aligned with individual sections of the fingers with a bottom row on the palm of the hand.
US-A-2007/279380 describes a computer input device that can be attached to a hand. The device includes an adjustable band that can be wrapped around the palm portion of the hand. Control buttons and a joystick are provided which are operated by the thumb and fingers of the hand to which the band is attached. The device communicates with a computer either wirelessly or via an attached cable.
US-A-2011/202306 describes an immersive virtual reality system in which one or more body sensor units are employed for interaction with the system. The body sensor units include gloves and vests which include sensing elements for determining the position of the relevant part of the user on which the gloves or vests are worn. A display unit is provided for viewing of virtual representations of parts of the body of the user in correct spatial relationship to his/her own view point.
Frati et al. in their article entitled “Using Kinect for hand tracking and rendering in wearable haptics”, World Haptics Conference (WHC), 2011, IEEE, IEEE, 21 Jun. 2011, pages 317 to 321, describe a wearable haptic device having motors for force feedback at finger pad contact points for position sensing within a field of view of a three-dimensional imaging system. A hand tracking algorithm is disclosed in which main hand points, such as fingertip positions, are extracted from a bounding box calculated from a depth map. However, the largest part of the hand must be visible for the hand tracking algorithm to work properly, in particular, several heuristics are used to define the identity of each of the fingers and these may fail if the whole or part of one or more fingers is occluded in the field of view of the three-dimensional imaging system.
More recently, gesture recognition technologies based on imaging sensing signal analysis have created interactive systems which allow the user to provide commands simply by using predetermined gestures to interact with the computerised system controlling and running such interactive systems. Such gesture recognition technologies do not make use of any devices worn by the user, and make use of either full body movement analysis, or hands only movement analysis.
However, when using either independent hardware-based controllers or natural gesture-based movements, there are limitations in the number of predefined interactions that can be performed. For a hardware remote controller device, it must, for example, be handled and it does not support finger tracking or reliable hand gesture recognition. Similarly, a full-body natural gesture recognition system does not allow a simple click action to be performed in the same way as a button activated by a finger on a remote controller would be as it requires a specific movement to be performed exclusively among some other movements. Moreover, when using gesture recognition systems, it is difficult to perform at the same time a simple click action and moving a hand around a control screen. One reason is that, a click action tends to be performed by either detecting a still pose of a hand for a predetermined duration at a specific location, or detecting a forward and backward movement along a predefined axis, for example, the Z or depth axis, within a predetermined period of time. It is therefore not possible to make use of one single hand to perform the two actions within the same time.
Furthermore, the combination of a natural gesture recognition system with an existing remote control device is limited as the existing apparatus has to be handled, introducing strong constraints in the kind and the number of gestures able to be performed and recognised.